Learning Objectives

After completing this lesson, you’ll be able to:

• Define schema, and explain how the SchemaScanner works in dynamic workflows.
• Identify the scenarios where the SchemaScanner is useful.
• Use a SchemaScanner to pass a schema feature to a dynamic writer.
• Apply a schema dynamically and modify it at the same time.

Instructions

In this lesson, you will:

• Scroll down to read the text below.
• Complete the exercise by following the steps.
• Complete the Quiz toward the bottom of the page.
• Click 'Next' to mark the lesson complete.

Resources

• Starting workspace
  • If your computer has FMEData, the file path is C:\FMEData\Workspaces\AdvancedReadingAndWriting\build-dynamic-workflows-with-the-schemascanner.fmw
• Complete workspace
  • C:\FMEData\Workspaces\AdvancedReadingAndWriting\build-dynamic-workflows-with-the-schemascanner-complete.fmw
• Cedar Cottage.csv
  • C:\FMEData\Resources\DynamicWorkflows\Data\Cedar Cottage.csv

Introduction

With the SchemaScanner, you can easily extract and manipulate the schema of your datasets, tackling dynamic workspace issues such as schema standardization and schema drift. How does it work? The SchemaScanner gives you a list attribute with attribute names and data types. Downstream in your workspace, you can use this list attribute to manipulate your schema and create flexible workflows. Instead of defining a fixed schema on your writers, you can use the schema from the list attribute to flexibly define the schema at runtime. Quality assurance and schema drift handling just got easier!

What is a Schema?

A schema, sometimes referred to as the "data model," can be described as the structure of a dataset or, more accurately, a formal definition of a dataset’s structure.

Each dataset has its unique schema, which includes feature types, permitted geometries, user-defined attributes, and other rules that define or restrict its content. However, for most users, the most critical aspects of a schema are attribute names and data types.

Using the SchemaScanner

The SchemaScanner processes features and retrieves their schema by scanning for the attribute name and its data type. It will either scan all features or just a specified number of them. There’s also the option to exclude attributes using a Regular Expression to ensure a clean schema output.

The resulting output is a new schema feature output via the <Schema> output port. This new feature is also assigned the special attribute and value fme_schema_handling = ‘schema_only’, which allows a dynamic writer to recognize it as a schema feature. If you wish to continue using the original input features, these are passed via the Output port.

For more technical information on the SchemaScanner, check out the documentation.

Why you might want to use the SchemaScanner in your workspace:

• You want to ensure your dynamic writer is receiving a valid schema
• You don’t know the schema of incoming data and want to make sure it meets specific standards before being used in a dynamic writer
• You want to modify the schema before it reaches the writer
• You want to expose the schema for validation purposes (checking for schema drift)

Key things to remember:

• When used in a dynamic workspace, schema features should be output from the SchemaScanner using the Output Schema Features Before Data parameter.
• Most schemas contain attributes you don’t need in your final output dataset, so use the Ignore Attributes Containing parameter.

Introduction

As we’ve seen, dynamic workflows can obtain their schema from multiple locations.

One of those locations is in the workspace itself, and the SchemaScanner transformer is a key tool in making that happen.

As the name suggests, the SchemaScanner scans incoming features and produces a schema. That schema can then be used in a dynamic writer and stored in a specific FME feature.

The schema produced by this transformer may be different from the reader schema due to processes in the workspace, such as attribute renaming, removal, or addition.

In this example workspace, attributes are being added and removed from the reader schema:

The SchemaScanner generates a new schema for the output, assigns it a name, and passes it to a dynamic writer. Notice how the schema feature passes from the SchemaScanner's Schema output port to the same writer input port as incoming data. This is a somewhat unusual pattern in FME, so it's worth noting.

The dynamic writer is set up to recognize incoming schema features and will make use of them:

Notice how the Schema Source is set to “Schema From Schema Features” to inform the writer from where the schema is to be obtained. Also, notice the parameter that defines the name of the schema. This handles the situation where the same writer is fed multiple schema features.

Now, FME will write the data to the output CSV dataset using the schema modified by the AttributeManager.

A Schema Feature stores information about a schema that can be passed to a writer. The information is stored in a list attribute called attribute{}. It contains attribute names and data types, stored as attribute{}.name and attribute{}.fme_data_type:

Schema features are generated by the SchemaScanner transformer, the FeatureReader transformer, the Schema (Any Format) reader, and even the AttributePivoter transformer!

The schema generated by the FeatureReader is a copy of the dataset schema.

Going back to the SchemaScanner, let’s look at the parameters:

The important things to note here are:

• The transformer will output the schema feature before any data features. This is vital. The writer must receive the schema feature before any data features that use it.
• The transformer will exclude any attributes that contain fme_ or multi_from the schema. It will also not include format attributes. We don't need FME or format attributes in the output.
• The transformer will ignore attributes with empty (or null) values, i.e., excluded from the output schema, just as if they’d been removed in the AttributeManager.
• The transformer detects dates in the incoming data. If this had not been set, it would interpret values such as 20220812 as numbers, not dates.

Exercise

Jennifer, a GIS Specialist, publishes a CSV of market opening hours. She wants the output to carry a single Hours column worked out from the Open and Close times, with those two source columns dropped. She does not want to redefine the writer schema by hand every time the source data changes.

In this exercise, you will:

• Use a SchemaScanner to pass a modified schema to a dynamic writer.
• Set a dynamic writer to take its schema from a schema feature rather than from the reader.

1) Generate the Workspace

Generating with Dynamic Schema gives you a schema-less translation to start from. Running it once before you change anything shows what the reader schema looks like on its own.

• Start FME Workbench 2026.2 or later.
• Select Build > Generate Workspace and configure the following parameters:
  • Reader > Format: CSV (Comma Separated Value)
  • Reader > Dataset: the Cedar Cottage.csv download, or C:\FMEData\Resources\DynamicWorkflows\Data\Cedar Cottage.csv
  • Writer > Format: CSV (Comma Separated Value)
  • Writer > Dataset: C:\FMEData\Output\Training
  • Workflow Options: Dynamic Schema

• Click OK.
• Click Run and inspect the output to see what the source data looks like.
  • Selecting View Written Data opens the Select Dataset to View dialog. Specify the format and dataset to see the data. The Tips at the end of this exercise explain why FME asks.

2) Create the Hours Attribute

We will create a new attribute called Hours, a combination of the Open and Close attributes. But then we'll need to find a way to add that to the writer schema.

• Add an AttributeManager between the reader feature type and the writer feature type.
• Create a new attribute named Hours and open the Text Editor for its value.
• Enter @Value(Open) - @Value(Close).
• Delete the Open attribute and the Close attribute.
• Move Hours above Open and Close, using the Up arrow to reorder.
  • If the deletions run first, Hours has nothing to work from and comes out empty.

• Re-run the workspace.
• Inspect the AttributeManager's Output port. Observe it has the new Hours attribute.
• Inspect the output data. Observe it still carries the old schema without Hours and missing values for Open and Close. This is because its schema source is the reader. That is exactly the problem the SchemaScanner solves.

3) Add a SchemaScanner

The SchemaScanner reads the features as they are at that point in the workspace and builds a schema feature from them. Sending that schema feature to the writer alongside the data is what makes the output follow the changes you just made.

• Add a SchemaScanner after the AttributeManager.
• Connect both the Output port and the <Schema> port to the writer feature type.
• Open the SchemaScanner and confirm Output Schema Before Data Features is set to Yes.
  • The writer has to receive the schema feature before any data features that rely on it.
• Click OK.

4) Set the Writer Schema Source

The writer still expects its schema from the reader. Pointing it at the schema feature instead is the step that hands control to the SchemaScanner.

• Open the parameters for the <Dynamic> CSV writer feature type.
• Click the ellipsis button beside Schema Sources, clear Cedar Cottage and select Schema From Schema Feature.

• Click OK.
• Set Schema Definition Name to fme_feature_type_name.
  • The SchemaScanner creates this attribute. Here it unhelpfully produces a file called CSV.csv. A static name would read better in this specific case, but where the input feature types carry real table names, fme_feature_type_name preserves them.

5) Run the Workspace

The output should now match what the workspace produces rather than what it read, which is the whole point of scanning the schema mid-workflow.

• Confirm your workspace matches the layout below.

• Click Run and inspect the output.
  • Open and Close are gone, and an Hours attribute has taken their place.

You have built a workspace whose output schema follows the data as the workspace changes it, rather than mirroring the source. If the incoming data changes, FME writes whatever schema it receives, and as long as Open and Close are present it will keep producing Hours in their place.